Several interactive display systems are currently known. For example, a user interface platform was developed in the MIT Media Lab, as reported by Brygg Ullmer and Hiroshi Ishii in “The metaDESK: Models and Prototypes for Tangible User Interfaces,” Proceedings of UIST 10/1997:14-17. This article describes how the metaDESK includes a near-horizontal graphical surface that is used to display two-dimensional (2D) geographical information. A computer vision system inside the desk unit (i.e., below the graphical surface) includes infrared (IR) lamps, an IR camera, a video camera, a video projector, and mirrors. The mirrors reflect the graphical image projected by the projector onto the underside of the graphical display surface to provide images that are visible to a user from above the graphical display surface. The article further teaches that the IR camera can detect passive objects called “phicons” that are placed on the graphical surface. Thus, when the IR camera detects the pattern applied to the undersurface of a “Great Dome phicon,” it responds by displaying a map of the MIT campus on the graphical surface, with the actual location of the Great Dome in the map positioned where the Great Dome phicon is located.
Moving the Great Dome phicon over the graphical surface manipulates the displayed map by rotating or translating the map in correspondence to the movement of the phicon by a user. Clearly, the IR vision-sensing system used in this prior art interactive display is able to detect objects like the phicon, based upon the pattern applied to it. There is no discussion of details involved in simply detecting an object without a pattern, or in determining a relative position of an object above the display surface.
A similar technique for sensing objects on a display surface is disclosed in several papers published by Jun Rekimoto of Sony Computer Science Laboratory, Inc., in collaboration with others. These papers briefly describe a “HoloWall” and a “HoloTable,” both of which use IR light to detect objects that are proximate to or in contact with a display panel on which a rear-projected image is visible. The rear-projection panel, which is vertical in the HoloWall and horizontal in the HoloTable, is semi-opaque and diffusive, so that objects reflecting IR light back through the panel become more clearly visible to an IR camera as they approach and then contact the panel. The objects thus detected can be a user's fingers or hand, or other objects. Again, these papers are generally silent regarding the process used for detecting an object based upon the IR light reflected from the object and also fail to discuss determining the relative separation between an object and the display surface.
Interaction between graphical objects on an interactive display surface and a three-dimensional object has generally been limited to treating each three-dimensional object as a discrete point. Part of the goal of this type of interface is to maintain a style of “direct manipulation” when manipulating virtual objects that is very similar to that experienced when manipulating real physical objects. A user's prior experience with manipulating real physical objects should thus inform the user's interaction with the corresponding virtualized objects. In the spirit of direct manipulation, the best interface is one which simulates as closely as possible the real physical movement of objects. For example, a user might initiate rotation of an object such an image of a photo by placing one finger in the upper-left corner and one finger in the lower-right corner of the virtual photo and using the movement of the fingers to drive the rotation of the image until the user removes the fingers from the surface of the virtual photo. As a further example, scaling might take place by placing two fingers (i.e., one on either side) on an image such as a photo or another object, near the mid-point between top and bottom and then moving the fingers outwardly. Translation might take place by placing a single digit near the center of the object and moving the digit in any direction. Unfortunately, there are several problems with this methodology. First, the user must be familiar with the specific modes of interaction with the interactive display interface, and these techniques for manipulating an object are not as intuitive as might be desired. Second, the interactive display system must employ heuristics to determine which of a number of different modes of manipulating an object that the user wishes to initiate when the user's fingers are placed on the interactive display surface in specific configurations. Third, this approach supposes that fingers may be reliably detected and tracked on the interactive displays surface.
In another approach for manipulating an object with objects such as the user's fingers, the digits might be detected and tracked when in contact with the interactive display surface. This approach avoids the triggering of specific modes based on the initial configuration of the fingers on the object being manipulated, thereby avoiding the first and second problems described above. Unfortunately, this approach also supposes that fingers may be reliably detected and tracked, which is currently problematic.